Smart detecting and feedback system for smart piano

ABSTRACT

A smart musical instrument system may include a musical instrument, a first sensor, a second sensor, a processor device, and a reminder device. The first sensor may be configured to obtain first performance data reflecting operations of the musical instrument. The may be second sensor configured to receive second performance data associated with hand posture of a user of the smart instrument system. The processor device may be in communication with the first sensor and the second sensor, and may be configured to generate feedback based on at least one of the first performance data or the second performance data. The reminder device may be configured to deliver the feedback to the user.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation of International Application No.PCT/CN2016/101794, filed on Oct. 11, 2016, the content of which isincorporated herein by reference.

TECHNICAL FIELD

This application relates to a smart piano system, and more particularly,to a smart detecting and feedback system and method for a smart piano.

BACKGROUND

A piano (e.g., an acoustic piano) is a musical instrument played using akeyboard. The piano may include a protective case, a soundboard, metalstrings, hammers, keys (e.g., white keys and black keys), and pedals.The keys may be mechanically connected to hammers. When a key of theacoustic piano is pressed by a player, a hammer may strike the metalstrings and the strings may vibrate at their respective resonantfrequency. These vibrations may be transmitted to the soundboard andamplified by coupling the acoustic energy to the air. A note produced bythe piano may be sustained, by application of one or more pedals of thepiano, even when the keys are released.

As one of the world's most popular musical instruments, the piano iswidely played and studied today. A piano student (especially a beginningstudent) may find it difficult to learn fingering and hand positions,fingering sequences, pedaling techniques, and other piano techniques.Accordingly, it is desirable to provide a smart detecting and feedbacksystem and method to assist in learning piano.

SUMMARY OF THE INVENTION

According to an aspect of the present disclosure, a smart musicalinstrument system may include a musical instrument, a first sensor, asecond sensor, a processor device, and a reminder device. The firstsensor may be configured to obtain first performance data reflectingoperations of a musical instrument. The second sensor may be configuredto receive second performance data associated with hand posture of auser of the smart instrument system. The processor device may be incommunication with the first sensor and the second sensor, and may beconfigured to generate feedback based on at least one of the firstperformance data or the second performance data. The reminder device maybe configured to deliver the feedback to the user.

According to another aspect of the present disclosure, a method foroperating a smart musical instrument system may include: obtaining, by afirst sensor connected to a musical instrument, first performance datareflecting operations of the musical instrument; obtaining, by a secondsensor connected to a user, second performance data associated with handposture of the user; generating feedback by a processor device incommunication with the first sensor and the second sensor based on atleast one of the first performance data or the second performance data;and delivering the feedback to user by a reminder device.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is further described in terms of exemplaryembodiments. These exemplary embodiments are described in detail withreference to the drawings. These embodiments are non-limiting exemplaryembodiments, in which like reference numerals represent similarstructures throughout the several views of the drawings, and wherein:

FIG. 1 is a block diagram illustrating an example of a smart detectingand feedback system 100 in accordance with some embodiments of thepresent disclosure;

FIG. 2 is a block diagram illustrating an example of smart piano systemaccording to some embodiments of the present disclosure;

FIG. 3 is a block diagram illustrating an example data analysis unit 205of the processing device 104 in accordance with some embodiments of thepresent disclosure;

FIG. 4 is a block diagram illustrating an example feedback module of asmart detecting and processing system in accordance with someembodiments of the present disclosure;

FIG. 5A is a flowchart illustrating a method for delivering a feedbackcontent to the user of the smart piano;

FIG. 5B is a flowchart illustrating a method for processing signal andgenerating a feedback content according to some embodiments of thepresent disclosure.

FIG. 6 is a diagram illustrating an embodiment of the presentdisclosure;

FIG. 7A and FIG. 7B are diagrams illustrating an embodiment of thepresent disclosure;

FIG. 8 is a diagram illustrating an example of detecting userinformation associated to user performance according to some embodimentsof the present disclosure;

FIG. 9A and FIG. 9B are diagrams illustrating two examples of detectinguser information associated to user performance according to someembodiments of the present disclosure;

FIG. 10 is a flowchart illustrating a process for a smart detecting andfeedback system according to some embodiments of the present disclosure;and

FIG. 11 is a flowchart illustrating a process for a smart detecting andfeedback system according to some embodiments of the present disclosure.

DETAILED DESCRIPTION

Subject matter will now be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of illustration, specific example embodiments.Subject matter may, however, be embodied in a variety of different formsand, therefore, covered or claimed subject matter is intended to beconstrued as not being limited to any example embodiments set forthherein; example embodiments are provided merely to be illustrative.Likewise, a reasonably broad scope for claimed or covered subject matteris intended. Among other things, for example, subject matter may beembodied as methods, devices, components, or systems. The followingdetailed description is, therefore, not intended to be limiting on thescope of what is claimed.

Throughout the specification and claims, terms may have nuanced meaningssuggested or implied in context beyond an explicitly stated meaning.Likewise, the phrase “in one embodiment” as used herein does notnecessarily refer to the same embodiment and the phrase “in anotherembodiment” as used herein does not necessarily refer to a differentembodiment. It is intended, for example, that claimed subject matterincludes combinations of example embodiments in whole or in part.

In general, terminology may be understood at least in part from usage incontext. For example, terms, such as “and”, “or”, or “and/or,” as usedherein may include a variety of meanings that may depend at least inpart upon the context in which such terms are used. Typically, “or” ifused to associate a list, such as A, B or C, is intended to mean A, B,and C, here used in the inclusive sense, as well as A, B or C, here usedin the exclusive sense. In addition, the term “one or more” as usedherein, depending at least in part upon context, may be used to describeany feature, structure, or characteristic in a singular sense or may beused to describe combinations of features, structures or characteristicsin a plural sense. Similarly, terms, such as “a,” “an,” or “the,” again,may be understood to convey a singular usage or to convey a pluralusage, depending at least in part upon context. In addition, the term“based on” may be understood as not necessarily intended to convey anexclusive set of factors and may, instead, allow for existence ofadditional factors not necessarily expressly described, again, dependingat least in part on context.

FIG. 1 illustrates an example of a smart detecting and feedback system100 in accordance with some embodiments of the present disclosure. Thesmart detecting and feedback system 100 may be configured to connect toa music instrument. The music instrument may include but is not limitedto one or more keyboard instruments, percussion, stringed instruments,or the like, or a combination thereof. For example, the keyboardinstrument may include but is not limited to a piano, an organ, a pianoaccordion, and/or an electronic keyboard. The percussion may include butis not limited to a xylophone, a celesta, a drum kit, and/or a drum. Thestringed instruments may include but is not limited to a violin, a harp,a guitar, an electric guitar, and/or bass. In some embodiments, themusic instrument may also include a smart piano.

The smart detecting and feedback system 100 may be configured to obtainand process user performance data and generate a feedback content touser based on user performance. As illustrated, the smart detecting andfeedback system 100 may include an identification module 101, adetecting module 102, a database 103, a processing device 104, and afeedback module 105.

Here and also throughout the present disclosure, a module may beimplemented in many different ways and as hardware, software or indifferent combinations of hardware and software. For example, all orparts of a module implementations may be a processing circuitry that mayinclude part or all of an instruction processor, such as a centralprocessing unit (CPU), a microcontroller, a microprocessor; or anapplication specific integrated circuit (ASIC), a digital signalprocessor (DSP), a digital signal processing device (DSPD), aprogrammable logic device (PLD), a field programmable gate array (FPGA),a controller, other electronic components; or as circuitry that includesdiscrete logic or other circuit components, including an analog circuitcomponent, a digital circuit component or both; or any combinationthereof. The circuitry may include discrete interconnected hardwarecomponents or may be combined on a single integrated circuit die,distributed among multiple integrated circuit dies, or implemented in aMultiple Chip Module (MCM) of multiple integrated circuit dies in acommon package, as examples.

The identification module 101 may be configured to identify a user whoaccesses the smart detecting and feedback system 100. The user may be astudent, or may be a teacher, or may be a musician intends to use thesmart detecting and feedback system 100. In some embodiments, the usermay have a registered account in the smart detecting and feedback system100 and have his or her personal information stored therein, so that theidentification module 101 may be able to identify the user when the useris trying to use the smart detecting and feedback system 100.

In some embodiments, for different user, the smart detecting andfeedback system 100 may identify user identity based on theidentification module 101. The smart detecting and feedback system 100may establish a database of personal information and generate personalfeedback content for each individual user. In some embodiments, the usermay use the smart detecting and feedback system 100 without theidentification module 101.

The identification module may identify the user through various means.In some embodiments, the user may use a mobile phone or other smartterminals to scan a two-dimension code to enter the smart detecting andfeedback system 100. In some embodiments, user may have user-specificportal authentication data to access the smart detecting and feedbacksystem 100, such as by visiting a webpage that the smart detecting andfeedback system 100 is associated with and inputting user ID and apassword through the webpage. In some embodiments, the smart detectingand feedback system 100 may include a fingerprint scanner and may storefingerprint information of an individual user, so that it may identifythe user by his or her fingerprint. Similarly, with a facial recognitioncircuit or bar code scanner, the smart detecting and feedback system 100may also identify a user by facial recognition or a bar code. The smartdetecting and feedback system 100 may also include any hardware orsoftware which may be used to identify a user. In some embodiments, auser may enter the smart detecting and feedback system 100 by a remoteserver connecting therewith and obtain user performance data of theuser.

The detecting module 102 may be configured to detect information relatedto user performance. The information may relate to or be used to assessa user performance. The information may include physiologicalinformation, a physical information, or the like, or a combinationthereof. For example, the physiological information may include useremotion, attitude, or the like, or any physiological information whichrelate to user performance. The physical information may include but isnot limited to speed, pressure, position, gesture, sound or the like, ora combination thereof. For example, when the instrument includes one ormore keys or strings, the physical information may also include but isnot limited to information about one or more keys or strings of theinstrument that are pressed during performance, timing information aboutthe keys or strings of the instrument (e.g., a time instantcorresponding to a position of a key or vibration status of a string ofthe instrument), one or more operation sequences of the keys or stringsof the instrument, timing information and musical notes producedaccording to motions of the keys/pedals or strings of the instrument. Insome embodiments, the user information may also include but is notlimited to information about learning style, learning receptivity,learning rate user playing custom, musical skill level, hobbies,experience, ability, or the like. The detecting module 102 may includeany suitable device that is capable of collecting user information,processing the user information, generating signals based on the userinformation, transmitting the signals to database 103 and the processingdevice 104 or performing any other function.

The detecting module 102 may include physical signaling deviceconfigured to detect physical information of the user, physiologicalsignaling device configured to detect physiological information of theuser, smart accessory, or any other detecting devices that may obtainuser information.

The physical signaling device may be configured to obtain physicalinformation, such as playing speed, key pressure or string vibration,key or string position, user gesture, sound or the like, or acombination thereof. The physical signaling device may include sensors,camera, microphone, or the like, which may obtain physical information.

The sensors may be but is not limited to, an accelerometer, a forcesensing resistor (FSR), sonar, an infrared (IR) sensor, a photo sensor,an inertia measurement unit (IMU) sensor, an Attitude/Heading ReferenceSystem (AHRS) sensor, a temperature sensor, or a general purpose motionor orientation sensor. In some embodiments, the sensor may be a sound(volume, power, or frequency) measurement device.

The microphone may include any suitable device which may detect soundand convert sound signal to electric signal. The microphone may beelectric type (moving coil, ribbon), capacitance type (DC polarization),piezoelectric type (crystal, ceramic), electromagnetic type, carbontype, semiconductor type, or the like, or a combination thereof. In someembodiments, the microphone may be a Micro Electrical-Mechanical System(MEMS) microphone.

The camera may include any suitable device which may obtain image andconvert image to electric signal. The camera may include but is notlimited to a video photometer, an infrared scanner, a video camera, orthe like.

The physiological signaling device may be configured to obtainphysiological information, such as user emotion, attitude and otherphysiological information. The physiological information may be assessedby obtaining brain wave, heart rate, micro-expression, blood pressure,pulse, or the like, or a combination thereof.

The smart accessory may include any suitable device which may obtainphysical information and physiological information or any other userinformation, such as a smart-band, a smart-ring, a smart-watch, a mobilephone, a smart-glove, a smart glass, or the like, or a combinationthereof. In some embodiments, the smart accessory may be configured asan accelerometer, a force sensing resistor (FSR), a sonar, an infrared(IR) sensor, a photo sensor, an inertial measurement unit (IMU) sensor,an Attitude/Heading Reference System (AHRS) sensor, a temperaturesensor, a general purpose motion, orientation sensor, or the like, or acombination thereof.

In some embodiments, the detecting module 102 may also perform functionsof processing, transmitting information, and displaying feedbackcontent.

The detecting module 102 may be connected to or communicate with thedatabase and the processing device 104 via a wireless connection or awired connection. For example, the detecting module 102 may transmit theinformation to the processing device 104 via Bluetooth.

The database 103 may be configured to store information, such as data,program, instructions, or the like, or a combination thereof. Theinformation may be provided by the identification module 101, thedetecting module 102, and the processing device 104, and the feedbackmodule 105.

In some embodiments, the database 103 may be configured to store userinformation obtained from the detecting module 102. In some embodiments,the database 103 may be configured to store the information about musicdata, such as music score, music teaching video, musician performance,or the like. In some embodiments, the database 103 may be configured tostore the data generated when the smart detecting and feedback system100 in working condition.

The database 103 may be any storage device. The database 103 may belocal, or remote. The database 103 may be but is not limited ahierarchical database, a network database, a relational database, or thelike, or a combination thereof. The database 120 may be a storage deviceto store information with electric energy, such as a variety of memory,a random access memory (RAM), a read only memory (ROM) and the like. Therandom access memory (RAM) may include but not limited to a decimalcounter, a select the number of tubes, a delay line memory, a Williamstube, a dynamic random access memory (DRAM), a static random accessmemory (SRAM), a thyristor random access memory (T-RAM), a zerocapacitor random access memory (Z-RAM) or a combination thereof. Theread only memory (ROM) may Include but not limited to a read-only memorybubble memory, a magnetic button line memory, a memory thin film, amagnetic plate line memory, a core memory, a magnetic drum memory, aCD-ROM drive, a hard disk, a magnetic tape, a nonvolatile memory early(the NVRAM), a phase change memory, a magnetoresistive random accessmemory module, a ferroelectric random access memory, a nonvolatile SRAM,a flash memory, a type of electronic erasing rewritable read-onlymemory, an erasable programmable read-only memory, a programmableread-only memory, a shielded heap read in combination with one or moreof memory, a floating connecting doors random access memory, a nanorandom access memory, a racetrack memory, a variable resistive memory, aprogrammable metallization cell and the like, or a combination thereof.The database 103 may be a storage device to store information withmagnetic energy such as a hard disk, a floppy disk, a magnetic tape, amagnetic a core memory, a bubble memory, an U disk, a flash memory orthe like, or a combination thereof. Database 103 may be a storage deviceto store information with optics such as a CD, DVD, or the like, or acombination thereof. Database 103 may be random access memory, theserial access storage, read-only memory or the like, or a combinationthereof. The database 103 may be a non-permanent memory, or a permanentstorage memory. The storage devices mentioned above are only someexamples, the database 103 in the working environment is not limitedthereof.

The processing device 104 may be configured to process and analyzeinformation, and generate a feedback content. The information may beobtained from the identification module 101, the detecting module 102,the database 103, the feedback module 105, or any device which may storeinformation. The processing device 104 may be a computational device andmay include additional functions of connecting or communicating with theidentification module 101, the detecting module 102, the database 103,the feedback module 105 or any device which may receive or transmitinformation.

In some embodiments, when a user play a piece of music, the processingdevice 104 may obtain an image information from camera, identify theimage, extract user posture information, and output a reminderinformation about a right posture. In some embodiments, the processingdevice 104 may include a function to calculate a music score to bedisplayed on a screen.

In some embodiments, the processing device 104 may be separated from amusic instrument. In such embodiments, the processing device 104 maycommunicate with the music instrument, via a short-range communication(e.g., Bluetooth), or via a wireless connection (e.g., WiFi), or via awired connection (e.g., universal serial bus or “USB”), or via someother connection types.

In some embodiments, the processing device 104 may be integrated withthe music instrument as a single unit. In some embodiments, theprocessing device may be a computer device which may be used as aprocessing device by installing software with the general purposecomputer. For example, the processing device 104 may be a tabletcomputer system that includes software configured to perform thefunctions of processing and analyzing information and generatingfeedback content.

In some embodiments, the processing device 104 may communicate with aremote server. The server may be cloud-based. The server may performfunctions of retrieving, processing, analyzing and storing information.The processing device 104 may obtain information from the server.

The feedback module 105 may be configured to present feedback contentrelated to user performance. The feedback content may includeinformation to inform and/or remind the user, in real-time while a useris playing a music, the difference between the user's performance and areference performance. For example, the feedback content may be of theform of audio content, video content, images, text, graphics, electricshock, vibration, or the like, or a combination thereof. The feedbackmodule 105 may obtain information from the processing device 104 andcommunicate with database 103 via a wireless connection or a wiredconnection.

The feedback module 105 may include but is not limited to displaydevices, printing devices, plotters, image output system, speech outputsystem, a magnetic recording apparatus or the like, or any device whichmay display, reminder, suggest, alert, teach user about performance. Insome embodiments, the feedback module 105 may play simultaneously inputsand outputs, for example, desktop computers, laptops, smart phones,tablet computers, personal digital assistants (personal digitalassistance, PDA), etc.

In some embodiments, the feedback module 105 may include a displaydevice that may present content related to the performance of a user,music played in the performance, another performance of the music,and/or any other information. In some embodiments, the size of a screenof the display device may be adapted to the size of a keyboard of thepiano. A virtual keyboard may be displayed on the screen. The virtualkeyboard can include multiple virtual keys corresponding to actual keysof the piano. In some embodiments, the number of the virtual keys andthe number of the actual keys may be the same. Information about one ormore operations to be performed on an actual piano key may be displayedin a portion of the screen including its corresponding virtual key. Suchinformation may include, for example, timing information related todepression and/or release of the actual piano key, a strength to beapplied to the actual piano key, a note duration, a note value, etc. Insome embodiments, various types of information may be presented on thescreen during the performance.

In some embodiments, the feedback module 105 may display how to play amusical note, a group of musical notes, or a portion of a music score,illustrate a musical notation, or point out a performance error made bya user playing the musical instrument.

In some embodiments, the feedback module 105 may provide a user a stagesummary. The stage summary may include information about user for aperiod learning experience, such as accustomed error, suggestion,evaluation, personalize lessons, user psychological state, or anyinformation related to user performance. As such, the feedback module105 may provide a user (e.g., a player, a piano student, etc.) withvisualized and personalized piano learning experience.

The different modules may be connected or communicate directly orindirectly with each other via a wireless connection or a wiredconnection. The wireless may be Bluetooth, WLAN, Wi-Fi, Zigbee, Z-Wave,EnOcean, infra-red data association (IrDA), Ultra Wideband (UWB), NearField Communication Services (NFC), mobile networks (2G, 3G or 4Gsignal), VPN, shared network, or the like, or a combination thereof. Thecommunication protocol may be Transmission Control Protocol (TCP), UserData Protocol (UDP), Internet Protocol (TP), Hypertext transfer Protocol(HTTP), File Transfer Protocol (FTP), Simple Mail Transfer protocol(SMTP), Post Office Protocl3 (POP3), or the like, or a combinationthereof. The wired connection may include but not limited to RS-232,CAN, TCP/IP, optical fiber, or the like, or a combination thereof.

The components of the smart detecting and feedback system that may beused in connection with the present system described above are notexhaustive and are not limiting. Numerous other changes, substitutions,variations, alterations, and modifications may be ascertained to oneskilled in the art and it is intended that the present disclosureencompass all such changes, substitutions, variations, alterations, andmodifications as falling within the scope of the present disclosure. Insome embodiments, the processing device 104 may be integrated with thefeedback module 105 as a single unit. These changes are still in thescope of the present disclosure.

FIG. 2 is a block diagram illustrating an example of smart piano systemaccording to some embodiments of the present disclosure. As shown inFIG. 2, the smart piano system may include a piano 200, a processingdevice 104, a feedback module 210 and an external component 209. Theprocessing device 104 may set up a connection with a cloud database 208.The processing device may further include a signal acquisition engine201, a storage engine 202, an external 203, a controller 204, a soundengine 206, and an auto-play engine 207. The controller may furtherinclude a data analysis unit 205. More or less components may beincluded in processing device 104 without loss of generality. Forexample, two or more of the modules may be combined into a singlemodule, or one of the module may be divided into two or more modules. Insome embodiments, one or more of the modules may reside on differentcomputing devices (e.g., desktops, laptops, mobile phones, tabletcomputers, wearable computing devices, etc.). In some embodiments, oneor more portions of the processing device 104 may be integrated with thepiano 200.

The piano 200 may be an acoustic piano, an electric piano, an electronicpiano, a digital piano, and/or any other musical instrument with akeyboard. A non-exclusive list of an acoustic piano that may be used inconnection with some embodiments of the present disclosure may include agrand piano, an upright piano, a square piano, a specialized piano (suchas a toy piano, a prepared piano, etc.), etc.

The signal acquisition engine 201 may obtain information about aperformance of a user (also referred to herein as “first performancedata”) according to one or more sensors integrated in the piano. Thefirst performance data may include but not limited to information abouta plurality of keys of piano 200 that are pressed during a userperformance, timing information associated with a plurality of keypressing (e.g. a time instant corresponding to a specific key or a timeinterval between two pressing of a specific key or keys.), sequences ofone or more operation of the keys, timing information about when a userstep on one or more pedals of the piano 200, and musical notes producedaccording to motions of the keys and/or the pedals of the piano 200. Thefirst performance data may be acquired by one or more sensors integratedin the piano. The first performance data may be generated based on keysignals and/or pedal signals acquired by the sensors. In someembodiments, the key signals and pedal signals may be an electric signalrepresented by a plurality of current value, a plurality of voltagevalue, or the like, or a combination thereof. When a user fingers on thekeyboard of the piano and/or feet step on the pedals, the firstperformance data may be recorded. The first performance data may bestored in the storage device 202 or any other storage device. The firstperformance data may also be obtained without the user fingering on thekeyboard. The user performance information may also be obtained by anexternal input engine 203 and it may be disclosed elsewhere in thepresent disclosure.

The storage device 202 may include any device that is capable of storingdata, such as data provided by the signal acquisition engine 201, theexternal input engine 203, the controller 204, or the like, or acombination thereof. Exemplary types of data that stored in the storageengine may include key signals and pedal signals acquired by the signalacquisition engine 201, first performance data generated by the signalacquisition engine 201, second performance data generated by theexternal input engine (As used herein, second performance data may referto user postures, face expressions, emotions, or the like, or acombination thereof, that detected by a plurality of externalcomponents.), and/or any other type of data that may be used toimplement a smart piano system in accordance with various embodiments ofthe present disclosure. The storage device 202 may include one or morecomponents. In some embodiments, the storage device 202 may include atleast one of a hard drive, a solid-state drive, a removable storagedrive (e.g., a flash memory disk drive, an optical disk drive, etc.), adigital video recorder, the like, or a combination thereof.

The external input engine 209 may be configured to connect the smartpiano system to a plurality of external components 209. As used herein,the external components 209 may include camera, finger ring, gloves, orsome other wearable things configured with sensors, or the like, or acombination thereof. In the external input engine 203, secondperformance data may be obtained and further transmitted into thestorage device 202. The type of the second performance data may bedetermined according to the sensors configured in the correspondingexternal components 209 connected with the external input engine 203.For instance, if the external components 209 include a camera, thesecond performance data may include the pictures and/or videosassociated with the hand posture when fingering the keys of the smartpiano, the face expressions while playing the smart piano, the posturesof the body, or the like, or a combination thereof. In some embodiments,the external components 209 may include a plurality of wearable devicesconfigured with a plurality of sensor that may detect user performancedata during playing the smart piano. For example, the external component209 is a plurality of finger rings with at least one sensors. The fingerring may be wore in a knuckle of a finger, or any other position thatmay reflect the actual movement of the finger. When pressing the keys ofthe smart piano, the sensors in the finger ring may record its positioninformation during the pressing, or accelerating information, orstrength information, or the like, or a combination thereof. Therecorded information may be transmitted into the external input engine203 and further stored in the storage device 202. In some embodiments,the second performance data may be brain wave during playing the smartpiano. For example, a helmet with brain wave detector may be wore by auser. The brain wave information may be transmitted into the externalinput engine 203 and for further analyzing.

The controller 204 may be configured to generate instructions for thesmart piano system. For example, the controller 204 may generateinstructions to instruct one or more components of the processing device104 to perform various functions, such as generating signals, receivingdata, processing data, storing data, transmitting data, presentingcontent (e.g., displaying video content and/images, playing audiocontent, etc.). The controller 204 may process data provided by thesignal acquisition engine 201, the storage device 202, the externalinput engine 203 or any other component of the smart piano system.

The controller 204 may comprise processor-based and/ormicroprocessor-based units. Merely by way of example, the processor mayinclude a microcontroller, a reduced instruction set computer (RISC),application specific integrated circuits (ASICs), anapplication-specific instruction-set processor (ASIP), a centralprocessing unit (CPU), a graphics processing unit (GPU), a physicsprocessing unit (PPU), a microcontroller unit, a digital signalprocessor (DSP), a field programmable gate array (FPGA), an acornreduced instruction set computing (RISC) machine (ARM), any othercircuit and/or processor capable of executing the functions describedherein, the like, or any combination thereof.

The controller 204 may further include a data analysis unit 205. Thedata analysis unit 205 may be configured to analyze the firstperformance and/or the second performance data, and some predeterminedinformation data set as default and stored in the storage device. Forexample, the hand posture information detected by an external component209 and further transmitted into the external input engine 203 may becompared with a pre-stored hand posture information in the storagedevice 202. Details of the data analysis unit 205 may be disclosedelsewhere in the present disclosure, for example in the description ofFIG. 3.

The sound engine 206 may include any device that is capable ofgenerating audio content representative of acoustic sounds or electricsounds, such as one or more synthesizers, sequencers, audio codecs,signal processors, speakers, etc. The sound engine 206 may be configuredto generate electric sounds and/or acoustic sounds. The acoustic soundsmay be generated by acoustic parts of the piano 200, such as keys,pedals, hammers, strings, and/or any other portion of the piano 200 thatcan generate an acoustic sound.

In some embodiments, the sound engine 206 may generate audio contentusing one or more synthesizers, sequencers, audio codecs, digitalprocessors, and/or any other device that can generate audio contentrepresentative of electric sounds. The audio content may be played backusing one or more speakers or any other device to produce electricsounds. The sound engine 206 can make various changes to the electricsounds. For instance, the electric sounds may be replaced or changedbased on the user's instructions.

The electric sounds may be played simultaneously with the acousticsounds or be played without the acoustic sounds. The electric sounds maybe also discontinuously played when the acoustic sounds are beingplayed. For instance, in one performance, the electric sounds may betriggered when a user starts to produce one or more selected musicalnotes. The electric sounds may be removed once the one or more selectedmusical notes have been played.

In some embodiments, the sound engine 206 can produce sounds based onmusical data provided by the external input engine 203, the storagedevice 202, and/or any other device. Merely by way of example, the smartpiano system may fetch musical data from a personal computer or a smartphone of the user. Then, the musical data may be converted into a formthat is recognizable by the sound engine 206 and the sound engine 206accordingly.

The auto-play engine 207 may be configured to automatically presentmusical data. The musical data may be obtained from the storage device202, and/or any other device. Merely by way of example, the musical datamay be generated according to a performance of the user, and stored inthe storage device 202. Exemplary musical data may include musicalmanuscript, one or more musical notes, timing of fingering one or morekeys, timing of operating one or more pedals, sequences of fingering oneor more keys, and sequences of operating one or more pedals, etc. Theauto-play engine 207 may generate sounds and/or images based on themusical data. Merely by way of example, after musical notes are playedby the user and stored as musical data in the storage device 202 by thesmart piano system, the auto-play engine 207 may automatically acquirethe musical data. The musical notes may then be generated with orwithout modifications. The auto-play engine 207 may also provide thesound engine 206 with the musical data and instructions for generatingsounds. Then the sound engine 206 may generate sounds accordingly.

The feedback module 210 may be configured to act as a connector betweenthe controller 205 and the external component 209. The controller 205may generate user readable information signal and transmitted into thefeedback module 210. As used herein, the user readable informationsignal may include signal of sound, image, vibration, electricity pulse,or the like, or a combination thereof. The type of the user readableinformation signal may be determined based on the type externalcomponent 209. For example, the plurality of the external components 209may include one or more cameras and one or more displays. The image ofthe hand posture or the other posture information of the user may bedetected by the one or more cameras. The image data may be thantransmitted and processed in the processing device 104. After theprocessing, the feedback module 210 may receive a signal relating to theimage detected before, and further transfer the signal to the one ormore display. In some embodiments, the plurality of the externalcomponents 209 may include one or more finger rings with sensor andvibration device. The one or more sensors configured with the fingerrings may detect the hand posture during playing. Then a signal relatingto the hand posture may be recorded and further transmitted andprocessed by the processing device 104. After the processing, thefeedback module 210 may receive a vibration signal relating to the handposture signal detected before and transmitted to the vibration device.The vibration device may then vibrate according to the vibration signal.

The processing device 104 may be connected to a cloud database 208according to some embodiments of the present disclosure. The storagedevice 202 may download and/or upload data with the cloud database 208.In some embodiments, if the storage device 202 receive a data from thesignal acquisition engine 201, or the external input engine 203, or thecontroller 204, it may send a copy version of the data to the clouddatabase 208 as a backup. In some embodiments the sound engine may 206receive information from cloud database 208. For example, if a userorder a musical note that not stored in the storage device 202, thesound engine may generate sound signal according to the musical note inthe cloud database 208. The controller 204 may receive information fromthe cloud database 208. In some embodiments, if a new type of externalcomponent is configured to the smart piano system, the controller 204may generate control instructions by downloading related instructionsfrom the cloud database 208. As used herein, the new type of externalcomponent 209 refer to a component that corresponding controlinstructions to it may not be stored or generated by the processingdevice 104. In some embodiments, the auto-play engine 207 may downloadauto-play information from the cloud database 208. The auto-playinformation may include timing of pressing the keys of the smart piano,the sequence of pressing the keys, the strength of pressing the keys,the time interval of pressing the keys, or the like, or a combinationthereof.

In some embodiments, each of the signal acquisition engine 201, thestorage device 202, the external input engine 203, the sound engine 206,the controller 204, and the auto-play engine 207 may include and/or beany of a general purpose device such as a computer or a special purposedevice such as a client, a server, and/or any other suitable device. Anyof these general or special purpose devices may include any suitablecomponents such as a hardware processor (which may be a microprocessor,digital signal processor, a controller, and/or any other suitablehardware processor), memory, communication interfaces, displaycontrollers, input devices, and/or any other suitable components. Forexample, each of the signal acquisition engine 201, the storage device202, the external input engine 203, the sound engine 206, the controller204, and the auto-play engine 207 may be implemented as or include apersonal computer, a tablet computer, a wearable computer, a multimediaterminal, a mobile telephone, a gaming device, a set-top box, atelevision, and/or any other suitable device. Moreover, each of thesignal acquisition engine 201, the storage device 202, the externalinput engine 203, the sound engine 206, the controller 204, and theauto-play engine 207 may comprise a storage device, which may include ahard drive, a solid state storage device, a removable storage device,and/or any other suitable storage device. Each of the signal acquisitionengine 201, the storage device 202, the external input engine 203, thesound engine 206, the controller 204, and the auto-play engine 207 maybe located at any suitable location. Each of the signal acquisitionengine 201, the storage device 202, the external input engine 203, thesound engine 206, the controller 204, and the auto-play engine 207 maybe implemented as a stand-alone device or integrated with one or moreother components of processing device 104.

The components of the smart piano system that may be used in connectionwith the present system described above are not exhaustive and are notlimiting. Numerous other changes, substitutions, variations,alterations, and modifications may be ascertained to one skilled in theart and it is intended that the present disclosure encompass all suchchanges, substitutions, variations, alterations, and modifications asfalling within the scope of the present disclosure.

FIG. 3 illustrates an example data analysis unit 205 of the processingdevice 104 in accordance with some embodiments of the presentdisclosure. The data analysis unit 205 may include but is not limited toa physiological signal processor 301, a physical signal processor 302.The data analysis unit 205 may process, analysis, calculate, retrieval,store, and transmit data. In some embodiments, the data analysis unit205 may also include a storage device. The physiological signalprocessor 301, the physical signal processor 302 may be connected orcommunicate with the signal acquisition engine 201, the storage device202, the external input engine 203, the controller 204, the sound engine206, the auto-play engine 207, the feedback module 105 via a wirelessconnection or a wired connection.

The physiological signal processor 301 may be configured to process,analysis, calculate, retrieval, store, transmit physiologicalinformation such as user emotion, attitude, or the like, or anyphysiological information which relate to user performance. In someembodiments, the physiological information may also include rain wave,heart rate, micro-expression, blood pressure, pulse, or the like, or acombination thereof, which may be used to assess a user psychologicalstate related to user performance.

The physical signal processor 302 may be configured to process,analysis, calculate, retrieval, store, and transmit physical informationsuch as playing speed, key pressure, user position, gesture, sound orthe like, or a combination.

The physiological signal processor 301 and the physical signal processor302 may be any commercially available or custom enterprise, application,personal, pervasive and/or embedded microprocessor, microcontroller,digital signal processor or the like. The physiological signal processor301 and the physical signal processor 302 may be local or remote.

The components of the data analysis unit 205 may be used in connectionwith the present system described above are not exhaustive and are notlimiting. Numerous other changes, substitutions, variations,alterations, and modifications may be ascertained to one skilled in theart and it is intended that the present disclosure encompass all suchchanges, substitutions, variations, alterations, and modifications asfalling within the scope of the present disclosure. In some embodiments,the data analysis unit 205 may be configured in the detecting device102. In some embodiments, the physiological signal processor 301 and thephysical signal processor 302 may be integrated into a single unit.These changes are still in the scope of the present disclosure.

FIG. 4 illustrates an example feedback module of a smart detecting andprocessing system in accordance with some embodiments of the presentdisclosure. A feedback module 105 may include but is not limited to areal-time output 401, a stage output 402, and a display 403. In someembodiments, the feedback module 105 may also include a storage device.The different units of real-time output 401, stage output 402, anddisplay 403 may be connected or communicate with each other via awireless connection or a wired connection.

The real-time output 401 may be configured to output a real-timefeedback content to the display 403 and/or an external component 209.The real-time feedback content may include any information when userplay a piece of music, for example, music score, performance video,error notes reminder, emotion, fingering, posture, or the like, acombination thereof. For example, the feedback content may be of avisual form displayed on a screen, such as textual description of thefeedback, a score, or graphical information (e.g., a progress bar),showing how accurate the user's performance compared to a referencemusical performance. The visual feedback content may reflect accuracy ofthe user's performance within a predetermined period of time, or anoverall accuracy since the beginning of the performance. The feedbackcontent may also be tactual. For example, through a wearable device,such as finger rings, the real-time output 401 may control may input anelectric shock or vibration to the user's particular finger or locationof the user's hand to inform and/or reminder the user when a differencebetween a musical note or rhyme corresponding to that finger or hand isgreater than a threshold value. In some embodiments, the real-timeoutput 401 may be configured to output electric shock, vibration to theexternal component 209 to reminder user error via the external component209.

The stage out-put 402 may be configured to output a stage feedbackcontent to the display 403 and an external component 209. The stagefeedback content may be a stage summary after a period time. The stagesummary may include information about user for a period of learningexperience, such as learning style, learning receptivity, learning rate,accustomed error, suggestion, evaluation, skill level, userpsychological state, or any information related to user performance.

In some embodiments, the stage out-put 402 may be configured to outputpersonalize education plan to user according to user performance.

In some embodiments, the real-time output 401 and the stage output 402may output the feedback content to the database 102, the cloud database208, or other storage devices, which may store information.

The display 403 may be configured to present feedback content to theuser. The display 403 may be a device which may present feedback contentto user. The display 403 may include but is not limited to a printingdevice, plotter, image output system, speech output system, magneticrecording apparatus, or the like, or a combination thereof.

In some embodiments, the display 403 may be an intelligent terminal forexample, a desktop computer, a laptop, a smart phone, a tablet computer,a personal digital assistant (PDA), or the like, or any device which mayreceive feedback content. In some embodiments, the display 403 may be amultimedia display which may display audio, image, text or the like, ora combination thereof.

In some embodiments, the display 403 may perform the functions of thedetecting module 102 such as a microphone may be used as a music outputdevice, or a music input.

The components of the feedback module 105 may be used in connection withthe present system described above are not exhaustive and are notlimiting. Numerous other changes, substitutions, variations,alterations, and modifications may be ascertained to one skilled in theart and it is intended that the present disclosure encompass all suchchanges, substitutions, variations, alterations, and modifications asfalling within the scope of the present disclosure. In some embodiments,the real-time output 401 may be integrated with the stage output 402 asa single unit. In some embodiments, the feedback may be configured witha processor to process information for outputting. In some embodiments,the real-time output 401 and the stage output 402 may be controlled bythe controller 204. These changes are still in the scope of the presentdisclosure.

FIG. 5A is flowchart illustrating a method of delivering a feedbackcontent to the user of the smart piano system as introduced in FIGS.1-4.

In step 501, the smart piano system may detect and/or obtain userinformation associated with user performance. The user information mayinclude first performance data and second performance data as introducedin FIGS. 1-4, or the like, or a combination thereof.

If the user information is detected and/or obtained by a sensorinstalled in the smart piano 200 and transmitted into the signalacquisition engine 201, the user information may be categorized as thefirst performance data. If the user information is detected and/orobtained by a sensor installed in the external component 209, the userinformation may be categorized as the second performance data.

In some embodiments, the user information may include the identity ofthe user. For example, the user may be required to type in the ID numberand/or the password via an input device (e.g., a key board, a remotecontroller, a touch panel, or the like, or a combination thereof.). Theperformance data may refer to user postures, hand postures, faceexpressions, emotions, pattern of brain wave, or the like, or acombination thereof. The user information associated with userperformance may be used to assess the performance of the user. Forinstance, if the hand posture of a user during a playing is similar to ahand posture file recorded in the storage device 202, the performance ofthe user may be considered as well. For another example, if the faceexpressions of the user keep concentrate during a playing, the user maybe considered to enjoy the playing. In some embodiments, the detectingmay be performed simultaneously with the user performance. Some of theuser information (e.g. the hand posture of the user) may need to bemonitored in real time. The assessment according to this kind of userinformation may need intact data during the playing. In someembodiments, the detecting may be performed at preset intervals. Somekind of the user information may not need to be monitored all the time.Such kind of user information may include face expression, brain wave,body shape, or the like, or a combination thereof. This kind of userinformation may be detected every several time intervals (e.g., everysecond). The time interval may be predetermined by the user, or set asdefault.

According to exemplary embodiment of the present disclosure, the userinformation may also include a plurality of historical feedback frompast performances of the user.

In step 502, the smart piano system may generate a signal according tothe user information. The type of the signal may include current value,voltage value, or the like, or a combination thereof. The signal may begenerated by the external component 209 configured with a plurality ofsensors. The intensity of the signal may be determined according to thekind of the corresponding sensor configured in the external component209. For example, if the external component 209 is a finger ring withone or more sensors record the position information of the correspondingfinger, the intensity of the signal may increase if the correspondingfinger pressing the key. In some embodiments, the external component 209is a camera, the intensity of the signal may not be changed since thecamera may monitoring the fingers all the time. The external component209 may also be a helmet with brain wave sensors. When the user isplaying a music on the piano, the brain wave sensors are configured tosense, detect, and collect brain wave of the user. Accordingly, thesecond performance data may also include brain wave information of theuser when operating the piano.

In step 503, the signal may be transmitted to a processor device 104. Asshown FIG. 2, the signal may be generated by the external component 209and further transmitted to the external input engine 203 of theprocessing device 104. The transmission of the signal from a pluralityof external components 209 to the external input engine 203 may bethrough wired transmission or wireless transmission. Exemplary wiredtransmission may include RS-232, CAN, TCP/IP, optical fiber, or thelike, or a combination thereof. Exemplary wireless transmission mayinclude Bluetooth, WLAN, Wi-Fi, Zigbee, Z-Wave, EnOcean, infra-red dataassociation (IrDA), Ultra Wideband (UWB), Near Field CommunicationServices (NFC), mobile networks (2G, 3G or 4G signal), VPN, sharednetwork, or the like, or a combination thereof. In some embodiments, thetransmission may be executed in real time during the playing. Forexample, a sequence of continuous image data may be generated by acamera, and a flux of image data may need to be transmitted to theprocessing device 104 in real time. In some embodiments, thetransmission may not need to be executed in real time during theplaying. For example, a plurality of position data may be recorded in astorage device of a finger ring. After the playing or a specific timeinterval, the position data may be transmitted to the processing device104 in a specific sequence according to the timing it be recorded.

In step 504, the smart piano system may process the signal and generatea feedback content accordingly.

As is shown in FIG. 2, the signal may be received by the external inputengine 203 and/or the signal acquisition engine 201, stored in thestorage device 202 and further transmitted to the controller 204. Insome embodiments, the signal may not need to be stored in the storagedevice, but transmitted to the controller 204 directly. The signal maybe analyzed by the data analysis unit 205. After the processing, afeedback content may be generated by the controller 204. A plurality ofinstructions may be generated according to the feedback content by thefeedback module 210. The processing method may be determined accordingto the type of the performance data detected before. Merely by way ofexample, the external components 209 may include a camera. The firstperformance data detected by the camera may be a sequence of imagesreflecting the changing of the user's hand posture during playing. Thesequence of images may be jointed together by the controller 204 togenerate a video content. In some embodiments, the external componentsmay include a position sensor. The first performance data detected bythe sensor may be the timing and position of a finger. The processingmethod may be to analyze the timing and position data of the finger togenerate a virtual hand similar to the users during playing. Thefeedback content may refer to some actions that may remind to user.Exemplary feedback content may include displaying images and/or videoson a display, driving a vibration device, driving an electricity pulsegenerator, or the like, or a combination thereof. The type of thefeedback content may be determined by a plurality of reminder devicesincluded in the external components 209. As used herein, the pluralityof reminder devices may refer to devices that may generate humansensible signals. Exemplary reminder devices may include display,vibration device, electricity pulse generator, or the like, or acombination thereof. For example, the feedback content may be of avisual form displayed on a screen, such as textual description of thefeedback, a score, or graphical information (e.g., a progress bar),showing how accurate the user's performance compared to a referencemusical performance. The visual feedback content may reflect accuracy ofthe user's performance within a predetermined period of time, or anoverall accuracy since the beginning of the performance. The feedbackcontent may also be tactual. For example, through a wearable device,such as finger rings, the real-time output 401 may control may input anelectric shock or vibration to the user's particular finger or locationof the user's hand to inform and/or reminder the user when a differencebetween a musical note or rhyme corresponding to that finger or hand isgreater than a threshold value.

According to embodiments of the present disclosure, the smart pianosystem may also process the plurality of historic feedback from pastperformances of the user. To this end, the data analyze unit may analyzethe user's past performance based on the plurality of historicalfeedback and generate a statistic result of the user performance. Theresult may reflect characteristics of the user in his/her performance.For example, the data analyze unit may generate a statistic resultshowing what note or a combination of notes that the user failed toperform correctly in the past. This result then may be delivered to theuser to show him/her where is her weak point in the performance andhighlight the improvements he/she would make. For example, thecharacteristics of the user performance may include a user performanceon the note or combination of notes that has a probability of mismatchto a reference performance greater than a predetermined probability.According to another embodiment, the statistic result may also reflectchanges of the characteristics of the user's performance over time. Forexample, the statistic result may show improvement of the user'sperformance in the above note or combination of notes over time, so thatwhen the result is delivered to the user, the user may be informed abouthis/her performance improvement.

In step 505, the smart piano system may deliver the feedback content tothe user.

In some embodiments, the plurality of instructions generated by thefeedback module 210 before may then be transmitted to the plurality ofexternal components 209 to drive corresponding reminder devices in theexternal components 209. The transmission of the instruction from thefeedback module 210 to the external components 209 may through wiredtransmission or wireless transmission.

Upon receiving the instructions, the corresponding reminder device inthe external components may generate human sensible signals. Forexample, the instruction may be a sequence of image data that may drivea display to show images on it. The instructions may be a pulse signalto drive a vibration device and/or an electricity pulse generator. Ifthe user pressing a wrong key or the hand posture is not correct, thevibration device may vibrate and/or the electricity pulse generator maygenerate an electricity pulse to remind the user.

According to embodiments, the reminder device may include an augmentreality device, such as a headset configured to display a virtualteacher teaching the user according to the feedback. When the user madean incorrect performance, such as a user performance that does not matcha reference performance, the virtual reality teacher may remind and/orcorrect the performance to the user.

It should be noted that the above description of the method ofdelivering a feedback content to the user of the smart piano is merelyprovided for the purpose of illustration, and not intended to limit thescope of the present disclosure. For person having ordinary skill in theart, multiple variations and modifications may be made under theteachings of the present disclosure. For example, the detecting methodand/or feedback content may be changed according to specificimplementation scenarios. Merely by way of example, some other externalcomponents may be included and the corresponding detecting and/orfeedback content may be changed correspondingly. In some embodiments,the feedback content may not be generated after processing the signalgenerated before, by the processor device. It may be generated directlyupon detecting the user performance associated with user performance.

FIG. 5B is a flowchart illustrating a method of processing the signalfrom the external input engine 203 and/or the signal acquisition engine201 and generating a feedback content according to some embodiments ofthe present disclosure. As mentioned before, the signal acquisitionengine 201 may receive signals (“the first signal”) carrying the firstperformance data from the piano 200. The external input engine 203 mayreceive signals (“the second signal”) carrying the second performancedata from the external component 209. The controller 204 may receive thefirst and/or second signals from the signal acquisition engine 201and/or the external input engine 203. The feedback content may begenerated based on one of the first signal and the second signal. Insome embodiments, the feedback content may be generated based on both ofthe first signal and the second signal.

In step 506, the controller 204 may obtain a first comparing result bycomparing the first signal transmitted from the signal acquisitionengine 201 with music data. The first signal transmitted from the signalacquisition engine 201 may include piano related information, forexample, a key identity information, a timing information of keypressing, a sequence information of a plurality of key pressing, atiming information of pedal stepping on, and musical note informationproduced by key and/or pedal motion. The music data may be a standarddata pre-stored in the storage device 202 and the same type as thesignal transmitted from the signal acquisition engine 201. In someembodiments, the first signal transmitted from the signal acquisitionengine 201 may contain a timing information of a key pressing by theuser. The corresponding music data may contain a standard timinginformation of the key. The timing information may be compared in thisstep.

In step 507, the controller 204 may access the first comparing result togenerate the feedback content. For example, if the timing informationmentioned before that contained in the signal transmitted from thesignal acquisition engine 201 matches or substantially matches thestandard timing information, the comparing result may be considered ascorrect (e.g., similarity between the timing information of the firstperformance data and that of the pre-stored standard timing informationis smaller than a threshold value). If the timing information mismatchesthe standard timing information, the comparing result may be consideredas incorrect. Feedback content may be generated based on the comparingresult. For example, if the comparing result is correct, no remindersmay be sent to the user, or a correct playing symbol may be displayed ona screen. If the comparing result is incorrect, a reminder informationmay be generated and sent to user via some reminder device.

In step 508, the controller 204 may calibrate the feedback contentgenerated before according to the second signal transmitted from theexternal input engine 203. In some embodiments, the timing informationcontained in the signal transmitted from the signal acquisition engine201 may not exactly represent the timing of a key pressing because ofsome delay of mechanical motion of the key or because of processingdelay of the signal acquisition engine 201. So the feedback content mayneed to be calibrated. In some embodiments, a camera may be used todetect the key pressing during playing. Another timing information ofkey pressing may be recorded by the camera and further transmitted tothe controller 204 via the external input engine 203. The timinginformation transmitted from the external input engine 203 may becompared with the timing information transmitted from the signalacquisition engine 201 and the standard timing information. Thecontroller 204 may obtain and/or generate a second comparing resultaccording to the comparing. The feedback content may be calibratedaccording to the second comparing result. Merely by way of example, anaverage value of timing information may be calculated based on thetiming information transmitted from the external input engine 203 andthe timing information transmitted from the signal acquisition engine201. The average value of timing information may be then compared withthe standard timing information to assess whether the key pressingmotion is correct.

It should be noted that the processes described above is provided forthe purposes of illustration, and not intended to limit the scope of thepresent disclosure. Apparently for persons having ordinary skills in theart, numerous variations and modifications may be conducted under theteaching of the present disclosure. However, those variations andmodifications may not depart the protecting scope of the presentdisclosure. In some embodiments, the comparing may be performed firstlybetween the signal transmitted from the external input engine 203 andthe standard timing information. And the signal transmitted from thesignal acquisition engine 201 may be used to calibrate the feedbackcontent. These are still in the protecting scope of present disclosure.

FIG. 6 is an embodiment of the present disclosure. As is shown in thefigure, a user is playing a smart piano 603. The smart piano 603 may beconfigured with at least a display 601 and a camera 600. To the smartpiano system, the display 601 and the camera 600 may be the externalcomponents 209. The display 604 may be set vertically relative to thekeyboard of the smart piano 603. In some embodiments, the display may beturned to a specific angle relative to the vertical position describedbefore to face to the user in an appropriate position. The specificangle may depend on the user's height and/or sitting posture. Forexample, a taller user may turn the display to a larger angle withrespect to the smart piano 603 in order for the display to face towardsthe user, whereas a shorter user may turn the display to a smaller anglewith respect to the user in order for the display to face towards theuser. At least one camera 600 may be installed on the smart piano 603.The position of the camera 600 may be set at the top and middle of thedisplay 604. In this case, the camera 600 may overlook the keyboardand/or the hand posture of the user. In some embodiments, the positionof the camera 600 may be set at one side of the display 600 or one sideof the smart piano. In this case, the camera 600 may capture a side viewof the keyboard and/or the hand posture of the user. The view angle ofthe camera 600 may be adjusted according to the demand of the user. Forexample, as is shown in the figure, the camera 600 may be pivotable todifferent directions. By pivoting the camera 600, a first view angle 601and a second view angle 602 may be acquired. The first view angle 601may monitor the keyboard and the hand posture of the user. The secondview angle 602 may monitor the face expressions of the user. Thedirection of the camera may depend on user demand. For example, when theuser wants to record the hand posture during playing, the view angle maybe adjusted to the first view angle 601. When the user wants to recordthe face expressions during playing, the view angle may be adjusted tothe second view angle 602. In some embodiments, the cameras 600installed on the smart piano 603 may be more than one. One of thecameras 600 may be set in a first position with the first view angle601. Another one of the cameras 600 may be set in a second position withthe second view angle 602. The plurality of cameras 600 may be operatedsimultaneously to record the hand posture and/or the face expressions ofthe user, or separately according instructions from a processing device104.

In this embodiment, the external components 209 may include at least onedisplay 604 and at least one camera 600. The user information associatedwith user performance may be detected by the camera 600. The userinformation may include hand posture, face expression, body posture, orthe like, or a combination thereof. The signal generated according tothe user information may be a plurality of image data, and/or a videodata. The image and/or video data may then be transmitted to a processordevice 104 of the smart piano. The image and/or video data may than beprocessed in the processor device 104. The processing may include imageand/or video reconstruction, image and/or video cutting, or the like, ora combination thereof. The processor device 104 may then generate afeedback content according to the processed image and/or video data. Thefeedback content may be a plurality of processed images and/or videos tobe displayed on the display 604 of the smart piano 603. Upon receivingthe feedback content, the display may show the plurality of processedimages and/or videos to the user.

FIG. 7A and FIG. 7B disclose an embodiment external component 209 of thepresent disclosure. According to the embodiment, the external component209 may be a plurality of finger rings 700. Each finger ring may includeone or more sensors and/or reminder devices to measure movements andpositions of the finger ring. Each finger ring of the plurality offinger rings 700 may be worn on a knuckle of the user. The position andmovement information of each the knuckles may be recorded by the fingerrings 700 and further used to reconstruct the hand posture of the user.In some embodiments, the finger rings 700 may be worn on the fingertipsof the user's hand. A sensor installed in the ring may detect thepressing speed of the key and further calculate the pressing strength onthe corresponding key.

FIG. 7A is a block diagram illustrating the devices in the finger ring700 according to some embodiments of the present disclosure. The fingerring 700 may include a microprocessor 701, an acceleration sensor 702, aposition sensor 703, a reminder unit 704, a storage 705, a communicationunit 706 and a power supply 707. The microprocessor 701 may be connectedwith the other devices via a data bus, or the like, or a combinationthereof. Merely by way of example, the processor may include amicrocontroller, a reduced instruction set computer (RISC), applicationspecific integrated circuits (ASICs), an application-specificinstruction-set processor (ASIP), a central processing unit (CPU), agraphics processing unit (GPU), a physics processing unit (PPU), amicrocontroller unit, a digital signal processor (DSP), a fieldprogrammable gate array (FPGA), an acorn reduced instruction setcomputing (RISC) machine (ARM), any other circuit and/or processorcapable of executing the functions described herein, the like, or anycombination thereof.

The Acceleration sensor 702 may include gyroscope, Hall sensor, magneticfield sensor, or the like or a combination thereof. The accelerationdata of the finger ring may represent the movement of the correspondingfinger. It may be further used to calculate of pressing strength of thefinger. The position sensor 703 may include electro-optical sensor, orthe like, or a combination thereof. The position sensor may record theposition of the finger ring at a specific timing. The position of thefinger ring may be an absolute position relative to the ground or thesmart piano, or may be a relative position relative to another fingerring. The reminder unit 704 may include the reminder device describedbefore, for example, a display, a vibration device, an electricity pulsegenerator, a LED, a speaker, or the like or a combination thereof. Thereminder unit 704 may be driven by instruction from the microprocessor701. The storage 705 may include any device that is capable of storingdata. The user performance data detected by the sensors may be stored inthe storage 705 temporarily and then transmitted to the microprocessor.The instructions generated by the microprocessor 701 may be stored inthe storage 705 temporarily and then transmitted to the reminder unit704. The communication unit 706 may be configured to set up atransmission channel between the finger ring 700 and the smart piano.The transmission may through wired transmission or wirelesstransmission. The power supply 707 may be configured to supplyelectricity power to the other devices in the finger ring 700.

In some embodiments, an acceleration data and/or a position data may bedetected by the acceleration sensor 702 and/or the position sensor 703.The acceleration data and the position data may refer to the userinformation associated with user performance. Then the data may beprocessed in the microprocessor 701 to generate a signal. The signal maybe then transmitted to the communication unit 706 for furthertransmission to the processing device 104 of a smart piano. Thecommunication unit 706 may set up a communication channel with theexternal input engine 203 of the processing device 104, and the signalmay be transmitted then. The communication unit 706 may then receive aninstruction from the feedback module 210 of the smart piano after thesignal transmitted before been processed. The instruction may betransmitted to the reminder unit 704 directly, or through themicroprocessor 701. The instruction may drive the corresponding reminderdevices in the reminder unit 704.

FIG. 8 is an example of detecting user information associated to userperformance according to some embodiments of the present disclosure. Theexternal components 209 may include a camera 801 and a plurality offinger ring 803. The camera may be set on the top of the smart piano, orsomewhere that may monitoring the keyboard 802 and the hand of the usersimultaneously. The finger ring 803 may include some or all of thedevices in the finger ring 700. The finger ring 803 may exhibitcharacteristic that may be detected by the camera 801. Exemplarycharacteristic may include color, shape, size, or the like, or acombination thereof. In some embodiments, the finger rings 803 worn ondifferent fingers may be coated with different color. The camera 801 maydetect the different colors of the finger rings 803. When the userpressing the key board 802, the movement of the color finger rings 803may represent the movement of corresponding fingers. The movement datamay be recorded by the camera 801 and further transmitted to theprocessor device 104 of the smart piano.

FIG. 9A and FIG. 9B are two examples of detecting user informationassociated to user performance according to some embodiments of thepresent disclosure. Mechanisms for detecting motions of a key of a pianousing a sensor is illustrated. The sensor may be integrated in the smartpiano to detect piano related information. As shown, it may include asensor 900. The sensor 900 (e.g., an opto-electric sensor) may include alight-emitting element 902 and a light-detecting element 903. Anexemplary list of the light-emitting element 902 may include visibleLED, laser LED, infrared LED, laser diode (LD), photocell, etc. Anexemplary list of the light-detecting element 903 may include phototube,active-pixel sensor (APS), bolometer, charge-coupled device (CCD),gaseous ionization detector, photoresistor, phototransistor, etc. Thelight-emitting element 902 may generate light of various wavelengths.For example, the light-emitting element 902 may generate visible light,infrared light, ultraviolet (UV) light, etc. In some embodiments, thewavelength of a beam of light emitted by the light-emitting element 902may be controlled by one or more motors using a Pulse Width Modulation(PWM) mechanism. The light-detecting element 903 may be configured toreceive the light and to convert it into an electronic signal (e.g., acurrent signal, a voltage signal, etc.).

Referring to FIG. 9A, in some embodiments, the light-emitting element902 and the light-detecting element 903 may be positioned under the key901. In some embodiments, a non-transparent extrusion, e.g., a plate904, may be mounted to a surface of the key 901. The non-transparentplate 904 may partially or completely prevent the light-detectingelement 903 from receiving the light emitted by the light-emittingelement 902. The non-transparent plate 904 may be mounted to a lowersurface of the key 901 (e.g., the bottom of the key 901). Thelight-emitting element 902 may constantly emit light towards thelight-detecting element 903. The light-emitting element 902 may alsodiscontinuously emit light towards the light-detecting element 903. Forinstance, there may be one or more time intervals between lightemissions. The one or more time intervals may be based on velocity ofthe user depressing the keys.

In some embodiments, a light beam 905 may be emitted by thelight-emitting element 902. When the key 901 is not pressed down, thekey stays at a “top” position. When a user presses the key 901, the keymay move downwards from the “top” position. When the key 901 does not gofurther, it reaches an “end” position. The non-transparent plate 904 maymove along with the key 901 and may block one or more portions of thelight beam 905. The amount of the light detected by the light-detectingelement 903 may vary due to the movement of the non-transparent plate904. For example, when the key 901 moves toward the “end” position, theamount of light detected by the light-detecting element 903 maydecrease. As another example, when the key 901 moves toward the “top”position, the amount of light detected by the light-detecting element903 may increase. The light-detecting element 903 can determineinformation about the amount of the received light over time and canconvert such information into one or more electronic signals (e.g., oneor more key signals).

Turning to FIG. 9B, in some embodiments, the non-transparent plate 904may be omitted from the smart piano system. For instance, thelight-emitting element 902 and the light-detecting element 903 may beplaced above or beneath the key 901, and the light beam 911 emitted bythe light-emitting element 902 may not be able to travel linearlytowards the light-detecting element 903. A light beam 911 emitted by thelight-emitting element 902 may be projected towards the key 901. Thelight beam 911 may be reflected by the key 901 once it reaches a surfaceof the key 901 (e.g., the upper surface, the bottom surface, etc.). Thereflected light 913 may then travel towards the light-detecting element903 and may be received by the light-detecting element 903. When a userpresses the key 901, the key may move downwards from the “top” positionto the “end” position. The distance that the light travels from thelight-emitting element 902 to the light-detecting element 903 may varydue to various motions of the key. The light-detecting element 903 maydetermine the time between light emission and light reception to recordthe change in distance that the light travels. The change in distancemay be converted into one or more electric signals by thelight-detecting element 903. Thus, the motions of the key may berecorded by the sensor 901.

The light-emitting elements and the light-detecting elements describedabove are not exhaustive and are not limiting. Numerous other changes,substitutions, variations, alterations, and modifications may beascertained to one skilled in the art and it is intended that thepresent disclosure encompass all such changes, substitutions,variations, alterations, and modifications as falling within the scopeof the present disclosure.

FIG. 10 is a flowchart illustrating a process for a smart detecting andfeedback system according to some embodiments of the present disclosure.The process may be implemented as a set of instruction in anon-transitory storage medium of the smart piano system 100, and may beexecuted by the processing device 104 of the smart piano system 100.

In step 1001, the processing device 104 may obtain performance data ofuser related to a performance of a piece of music. The performance datamay include information about motion of a key, motion of a pedal duringthe performance, positions of the keys, timing information about themotion of the keys (e.g., a time instant corresponding to depression ofthe key, a time instant corresponding to release of the key, a timeinstant corresponding to a position of the key, etc.), a depressionstrength, velocities of the key during its motion, and/or numbers of keygroups or numbers of each individual key, the time player operating onthe pedals, and numbers of each individual pedal or numbers of pedalgroups. The performance data may be obtained by the signal acquisitionengine 201 and/or the external input engine 203. The signal acquisitionengine 201 and/or the external input engine 203 may be connected orcommunicate with the detecting module 102 and database 103 via awireless connection or a wired connection.

In some embodiments, the performance data may also include user emotiondata, attitude data or the like. The emotion data may be used to assessuser performance with suitable emotional expression. The attitude datamay be used to assess user enthusiasm for playing musical instrument.

In some embodiments, the processing device 104 may also obtain userinformation related user performance, such as user playing custom,musical skill level, hobbies, experience, ability, or the like. The userinformation may be used to assess user performance. The user informationmay be stored in the database 103. The user information may be stored inthe cloud database 208.

In step 1002, the processing device 104 may obtain a music data asreference music data. The music data may be received from the database103, the external input engine 203, the storage device 202, a clouddatabase, a server, and/or any other device that is capable of providingmusic data. The music data may include any information about a piece ofmusic, such as music score, musician information, piano teaching video,piano teaching reference, fingering, musician performance, or the like.The music data may be an audio file, an image, a text, a video or thelike, or a combination thereof.

In some embodiments, the piece of music may be identified by a user(e.g., a player of the piano). For example, the smart detecting andfeedback system 100 may prompt the user to select a music score from alist of pieces of music score and/or to provide information about thepiece of music (e.g., by providing one or more user interfaces). Asanother example, the smart detecting and feedback system 100 mayretrieval and identify the music data such as music score by initiate adownload of the music data and/or extraction of the music data from adevice that stores the music data. In some embodiments, the piece ofmusic may be identified based on audio content related to a performanceof the piece of music (e.g., using suitable audio matching, fingering,and processing techniques).

In some embodiments, the processing device 104 may perform functions ofretrieving music data in the database 103, cloud database 208, storagedevice 202, or webpage, or the like.

In some embodiments, the processing device 104 may also performfunctions of analyzing performance data. For example, the image data ofuser performance may be used to identify image and extract the featuredata such as user micro-expressions, user posture, user fingering or anyfeature related to user performance.

In step 1003, the music data (i.e., the reference music data) may becompared with the performance data. In some embodiments, the processingdevice 104 may be used to compare performance data for a plurality ofperformed musical notes with model music data for a plurality of modelmusic notes. The model music data may be obtained through processing themusic data. In each alternative musical note match lineup, each modelmusical notes is matched with one of the performed musical notes to forma pair unless a corresponding performed musical note has been identifiedas missing. The alternative musical note match lineups are differentfrom one another by at least one pair of matching model musical note andperformed musical note. For each alternative musical note match lineup,the processing device 105 may analyze a set of note deviations, eachnote deviation being calculated according to the data for one of themodel musical notes and the data for the matching performed musicalnote. For each alternative musical note match lineup, the processingdevice 105 may analyze an overall performance deviation of theperformance music data from the model music data.

In step 1004, the processing device 1104 may assess the user performanceto generate a feedback content. The processing device 1104 may assessthe user performance based on comparing music data and performance data.In some embodiments, the processing device 1104 may assess based on userinformation.

In some embodiments, the feedback content may include learning sampleperformances from experts, paying more efforts on playing certainmusical notes, etc. In some embodiments, the feedback content may besuggestions from a cloud-based storage device, a server, and/or anyother device capable of providing the suggestions. For example, thesuggestions may be stored in a cloud-based storage device indexed by anypossible error. Each error may be directed to one or more suggestions.In some embodiments, the suggestions may be from one or more real-timeexpert databases. Each real-time expert database may be connected to oneor more experts which may provide suggestions after receiving one ormore errors. The smart detecting and feedback system may provide theerrors to real-time expert databases and extract suggestions from thereal-time expert databases. In some embodiments, the suggestions may begenerated based on history of errors recorded by the smart detecting andfeedback system. For example, if number of certain errors exceeds athreshold value, the smart detecting and feedback system may providesuggestions to prompt the user's notice on the error(s).

In step 1005: the processing device 104 may transmit the result to afeedback module 105. The feedback result may include audio content,video content, images, text, graphics, electric shock, vibration, or thelike, or a combination thereof. In some embodiments, the feedback module105 may display how to play a musical note, a group of musical notes, ora portion of a music score, illustrate a musical notation, or point outa performance error made by a user playing the musical instrument.

The processing device 104 may transmit the result to the feedback via awireless connection or a wired connection. The wireless connection maybe Bluetooth, WLAN, Wi-Fi, Zigbee, Z-Wave, EnOcean, infra-red dataassociation (IrDA), Ultra Wideband (UWB), Near Field CommunicationServices (NFC), mobile networks (2G, 3G or 4G signal), VPN, sharednetwork, or the like, or a combination thereof.

It is understood that each flowchart illustrations may be implemented bycomputer program instructions. These computer program instructions maybe provided to a controller of a general purpose computer, specialpurpose computer, and/or other programmable data processing apparatus toproduce the instructions, which execute via the controller of thecomputer and/or other programmable data processing apparatus, createmeans for implementing the functions specified in the flowchart.

It should be noted that the processes described above is provided forthe purposes of illustration, and not intended to limit the scope of thepresent disclosure. Apparently for persons having ordinary skills in theart, numerous variations and modifications may be conducted under theteaching of the present disclosure. However, those variations andmodifications may not depart the protecting scope of the presentdisclosure. In some embodiments, the detecting module 101 may beconfigured a microprocessor for processing data. In some embodiments,the different module combined with each other, these are still in theprotecting scope of present disclosure.

FIG. 11 is a flowchart illustrating a process for a smart detecting andfeedback system according to some embodiments of the present disclosure.The process may be implemented as a set of instruction in anon-transitory storage medium of the smart piano system 100, and may beexecuted by the processing device 104 of the smart piano system 100.

In step 1101, a performance data may be obtained by at least one sensor.The sensors may be but is not limited to, an accelerometer, a forcesensing resistor (FSR), sonar, an infrared (IR) sensor, a photo sensor,an inertial measurement unit (IMU) sensor, an Attitude/Heading ReferenceSystem (AHRS) sensor, a temperature sensor, or a general purpose motionor orientation sensor. In some embodiments, the sensor may be a sound(volume, power, or frequency) measurement device.

In some embodiments, a pressure sensor may be configured under thekeyboard and may be connected to the processing device 104 via awireless connection or a wired connection. When the user touches thekey, the pressure exerted on the key may be detected by the pressuresensor. The pressure sensor may capture the value of the pressure andgenerate a pressure signal. The pressure signal may be obtained by thesignal acquisition engine 201 and/or the external input engine 203 andtransferred to the controller 204 and data analysis unit 205.

In some embodiments, the sensor may be configured in a smart accessorysuch as smart finger ring, a smart wristband, a smart glove, a pair ofsmart glasses, or the like.

In step 1102, the performance data may be transmitted to a processor. Insome embodiments, the processor may be configured in the detectingmodule 101. In some embodiments, the processor may be configured in theprocessing device 104. The sensor may communicate or be connected withthe processor via wireless Bluetooth, WLAN, Wi-Fi, Zigbee, Z-Wave,EnOcean, infra-red data association (IrDA), Ultra Wideband (UWB), NearField Communication Services (NFC), mobile networks (2G, 3G or 4Gsignal), VPN, shared network, or the like, or a combination thereof.

In step 1103, the processor may compare the performance data with amusic score.

In some embodiments, the performance data may be analyzed by the dataanalysis unit 205 before compared with the music score. In someembodiments, a photo sensor may be configured to obtain a picture offinger position on a keyboard when user playing a piece of a music. Thephoto sensor may convert the optics signal to an electric signal, andtransfer the electric signal to the data analysis unit 205. The analysisunit 205 may identify the picture and capture the finger position on thekeyboard.

The data analysis unit 205 may also obtain a music score from thedatabase 102. In some embodiments, the music score may be retrieved fromthe database 102 by user. In some embodiments, the music score may beretrieved from a webpage and downloaded by user voice controlled.

In some embodiments, the feedback module 105 may display the music scoreon the screen, and even display a visual keyboard. The motion of thevisual key may be accordance with the music score.

In step 1104 and 1105, if the performance data matches the music score,user may continue to play next note. In some embodiments, the pressuresensor may obtain the motion of a key, and determine the note played bythe user. The processing device 104 may determine notes which should beplayed according to the music score, and compare to played note. If theplayed note matches with the music score, the user may be allowed tocontinue to play the next note. If the played note does not match withthe music note, the system may provide a reminder to hinder the user tocontinue to play next note.

In some embodiments, the smart detecting and feedback system 100 may beconfigured to implement an error detection mode and/or an errordetection module. For example, the error detection mode and/or modulemay detect errors in the performance information. The errors in theperformance information may include wrong timing of operating one ormore keys and/or pedals, wrong operation sequences of one or more keysand/or pedals, inappropriate strength of operating one or more keysand/or pedals, etc. If the user makes an error in the performance of apiece of a music, the system 100 may real-time detect the error, andhinder user to continue to play the piece of music. The smart detectingand feedback system 100 may also be capable of providing suggestions forthe player. The suggestions may be based on the errors in theperformance information. For example, the suggestions may includelearning a sample performance from another expert, paying more effortson playing certain musical notes, etc.

In some embodiments, the system may be configured with an externalcomponent in the music instrument to hinder the music instrument workingwhen user play a wrong error note. In some embodiments, the system mayreminder user via electronic shock, vibration. In some embodiments, thesystem may display the error on the screen or via sound to suggest userwhich note should be played.

It should be noted that the components of the controller provided aboveare provided for the purposes of illustration, and not intended to limitthe scope of the present disclosure. Apparently for persons havingordinary skills in the art, numerous variations and modifications may beconducted under the teaching of the present disclosure. However, thosevariations and modifications may not depart the protecting scope of thepresent disclosure.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, as apparent from the followingdiscussion, it is appreciated that throughout the description,discussions utilizing terms such as “sending,” “receiving,”“generating,” “providing,” “calculating,” “executing,” “storing,”“producing,” “determining,” “obtaining,” “calibrating,” “recording,” orthe like, refer to the action and processes of a computer system, orsimilar electronic computing device, that manipulates and transformsdata represented as physical (electronic) quantities within the computersystem's registers and memories into other data similarly represented asphysical quantities within the computer system memories or registers orother such information storage, transmission or display devices.

The terms “first,” “second,” “third,” “fourth,” etc. as used herein aremeant as labels to distinguish among different elements and may notnecessarily have an ordinal meaning according to their numericaldesignation.

In some implementations, any suitable computer readable media may beused for storing instructions for performing the processes describedherein. For example, in some implementations, computer readable mediacan be transitory or non-transitory. For example, non-transitorycomputer readable media can include media such as magnetic media (suchas hard disks, floppy disks, etc.), optical media (such as compactdiscs, digital video discs, Blu-ray discs, etc.), semiconductor media(such as flash memory, electrically programmable read only memory(EPROM), electrically erasable programmable read only memory (EEPROM),etc.), any suitable media that is not fleeting or devoid of anysemblance of permanence during transmission, and/or any suitabletangible media. As another example, transitory computer readable mediacan include signals on networks, in connectors, conductors, opticalfibers, circuits, any suitable media that is fleeting and devoid of anysemblance of permanence during transmission, and/or any suitableintangible media.

It should be noted that the piano equipped with the smart piano systemin some specific embodiments is provided for the purposes ofillustration, and not intended to limit the scope of the presentdisclosure. Apparently for persons having ordinary skills in the art,numerous variations and modifications may be conducted under theteaching of the present disclosure. However, those variations andmodifications may not depart the protecting scope of the presentdisclosure.

Furthermore, the recited order of processing elements or sequences, orthe use of numbers, letters, or other designations therefore, is notintended to limit the claimed processes and methods to any order exceptas may be specified in the claims. Although the above disclosurediscusses through various examples what is currently considered to be avariety of useful embodiments of the disclosure, it is to be understoodthat such detail is solely for that purpose, and that the presentdisclosure are not limited to the disclosed embodiments, but, on thecontrary, are intended to cover modifications and equivalentarrangements that are within the spirit and scope of the disclosedembodiments. For example, although the implementation of variouscomponents described above may be embodied in a hardware device, it mayalso be implemented as a software only solution—e.g., an installation onan existing server or mobile device.

Similarly, it should be appreciated that in the foregoing description ofembodiments of the present disclosure, various features are sometimesgrouped together in a single embodiment, figure, or description thereoffor the purpose of streamlining the disclosure aiding in theunderstanding of one or more of the various inventive embodiments. Thismethod of disclosure, however, is not to be interpreted as reflecting anintention that the claimed subject matter requires more features thanare expressly recited in each claim. Rather, inventive embodiments liein less than all features of a single foregoing disclosed embodiment.

What is claimed is:
 1. A smart musical instrument system comprising: afirst sensor configured to obtain first performance data reflectingoperations of a musical instrument; a second sensor configured toreceive second performance data associated with hand posture of a userof the smart instrument system; a processor device in communication withthe first sensor and the second sensor, configured to generate feedbackbased on at least one of the first performance data or the secondperformance data; and a reminder device configured to deliver thefeedback to the user.
 2. The smart musical instrument system of claim 1,wherein the processor device is further configured to: obtain aplurality of historical feedback from past performances of the user;generate a statistic result reflecting characteristics of the userperformance based on the plurality of historical feedback; output aperformance instruction to the user based on the statistic result. 3.The smart musical instrument system of claim 1, the characteristics ofthe user performance includes a user performance on the musicalinstrument that has a probability of mismatch to a reference performancegreater than a predetermined probability.
 4. The smart musicalinstrument system of claim 1, wherein the hand posture comprises fingerposture of a hand of the user; and the second sensor includes at leastone of: a plurality of finger rings, each finger ring is configured tobe worn on a finger of the hand and to measure at least one of aposition, a motion, or force of the finger; or a camera configured toobtain the hand posture image of the user.
 5. The smart musicalinstrument system of claim 1, wherein the second performance data isfurther associated with at least one of body posture or facialexpression of the user; and the second sensor includes a cameraconfigured to obtain the at least one of the body posture or the facialexpression of the user.
 6. The smart musical instrument system of claim1, wherein the second sensor includes a brain wave detector configuredto measure brain waves of the user; and the second performance datafurther includes brain wave information of the user when operating themusical instrument.
 7. The smart musical instrument system of claim 1,wherein the musical instrument includes at least one of a plurality ofkeys or a pedal; and the first performance data includes at least one ofa key identity information, a timing information of key pressing, asequence information of a plurality of key pressing, a timinginformation of pedal stepping on, and a musical note informationproduced by key, or a musical note information produced by pedal motion.8. The smart musical instrument system of claim 1, wherein to generatethe feedback the processor device is further configured to: compare thefirst performance data with a reference music data to generate afeedback content; and calibrate the feedback content according to thesecond performance data.
 9. The smart musical instrument system of claim1, wherein to generate the feedback the processor device is furtherconfigured to: compare the second performance data with a referencemusic data to generate a feedback content; and calibrate the feedbackcontent according to the first performance data.
 10. The smart musicalinstrument system of claim 1, wherein the reminder device includes anaugment reality headset configured to display a virtual teacher teachingthe user according to the feedback.
 11. A method for operating a smartmusical instrument system comprising: obtaining, by a first sensorconnected to a musical instrument, first performance data reflectingoperations of the musical instrument; obtaining, by a second sensorconnected to a user, second performance data associated with handposture of the user; generating feedback by a processor device incommunication with the first sensor and the second sensor based on atleast one of the first performance data or the second performance data;and delivering the feedback to user by a reminder device.
 12. The methodof claim 11, further comprises: obtaining, by the processor device, aplurality of historical feedback from past performances of the user;generating, by the processor device, a statistic result reflectingcharacteristics of the user performance based on the plurality ofhistorical feedback; outputting, by the smart musical instrument system,a performance instruction to the user based on the statistic result. 13.The method of claim 11, wherein the characteristics of the userperformance includes a user performance on the musical instrument thathas a probability of mismatch to a reference performance greater than apredetermined probability.
 14. The method of claim 11, wherein the handposture comprises finger posture of a hand of the user.
 15. The methodof claim 11, wherein the second performance data is further associatedwith at least one of body posture or facial expression of the user. 16.The method of claim 11, wherein the second performance data furtherincludes brain wave information of the user when operating the musicalinstrument.
 17. The method of claim 11, wherein the musical instrumentincludes at least one of a plurality of keys or a pedal; and the firstperformance data includes at least one of a key identity information, atiming information of key pressing, a sequence information of aplurality of key pressing, a timing information of pedal stepping on, amusical note information produced by key, or a musical note informationproduced by pedal motion.
 18. The method of claim 11, wherein thegenerating of the feedback further includes: comparing the firstperformance data with a reference music data to generate a feedbackcontent; and calibrating the feedback content according to the secondperformance data.
 19. The method of claim 11, wherein the generating ofthe feedback further includes: comparing the second performance datawith a reference music data to generate a feedback content; andcalibrating the feedback content according to the first performancedata.
 20. The method of claim 11, wherein the delivering of feedbackincludes displaying an augment reality teacher to the user to teach theuser according to the feedback.